A common scenario of Multilingual Neural Machine Translation (MNMT) is that each translation task arrives in a sequential manner, and the training data of previous tasks is unavailable. In this scenario, the current methods suffer heavily from catastrophic forgetting (CF). To alleviate the CF, we investigate knowledge distillation based life-long learning methods. Specifically, in one-tomany scenario, we propose a multilingual distillation method to make the new model (student) jointly learn multilingual output from old model (teacher) and new task. In many-to one scenario, we find that direct distillation faces the extreme partial distillation problem, and we propose two different methods to address it: pseudo input distillation and reverse teacher distillation. The experimental results on twelve translation tasks show that the proposed methods can better consolidate the previous knowledge and sharply alleviate the CF.

Large language models (LLMs) have demonstrated excellent zero-shot generalization to new language tasks. However, effective utilization of LLMs for zero-shot visual question-answering (VQA) remains challenging, primarily due to the modality disconnection and task disconnection between LLM and VQA task. End-to-end training on vision and language data may bridge the disconnections, but is inflexible and computationally expensive. To address this issue, we propose \emph{Img2Prompt}, a plug-and-play module that provides the prompts that can bridge the aforementioned modality and task disconnections, so that LLMs can perform zero-shot VQA tasks without end-to-end training. In order to provide such prompts, we further employ LLM-agnostic models to provide prompts that can describe image content and self-constructed question-answer pairs, which can effectively guide LLM to perform zero-shot VQA tasks. Img2Prompt offers the following benefits: 1) It can flexibly work with various LLMs to perform VQA. 2)~Without the needing of end-to-end training, it significantly reduces the cost of deploying LLM for zero-shot VQA tasks. 3) It achieves comparable or better performance than methods relying on end-to-end training. For example, we outperform Flamingo~\cite{Deepmind:Flamingo2022} by 5.6\% on VQAv2. On the challenging A-OKVQA dataset, our method even outperforms few-shot methods by as much as 20\%.

Data heterogeneity across clients in federated learning (FL) settings is a widely acknowledged challenge. In response, personalized federated learning (PFL) emerged as a framework to curate local models for clients' tasks. In PFL, a common strategy is to develop local and global models jointly - the global model (for generalization) informs the local models, and the local models (for personalization) are aggregated to update the global model. A key observation is that if we can improve the generalization ability of local models, then we can improve the generalization of global models, which in turn builds better personalized models. In this work, we consider class imbalance, an overlooked type of data heterogeneity, in the classification setting. We propose FedNH, a novel method that improves the local models' performance for both personalization and generalization by combining the uniformity and semantics of class prototypes. FedNH initially distributes class prototypes uniformly in the latent space and smoothly infuses the class semantics into class prototypes. We show that imposing uniformity helps to combat prototype collapse while infusing class semantics improves local models. Extensive experiments were conducted on popular classification datasets under the cross-device setting. Our results demonstrate the effectiveness and stability of our method over recent works.

We introduce BotSIM, a modular, open-source Bot SIMulation environment with dialog generation, user simulation and conversation analytics capabilities. BotSIM aims to serve as a one-stop solution for large-scale data-efficient end-to-end evaluation, diagnosis and remediation of commercial task-oriented dialog (TOD) systems to significantly accelerate commercial bot development and evaluation, reduce cost and time-to-market. BotSIM adopts a layered design comprising the infrastructure layer, the adaptor layer and the application layer. The infrastructure layer hosts key models and components to support BotSIM's major functionalities via a streamlined "generation-simulation-remediation" pipeline. The adaptor layer is used to extend BotSIM to accommodate new bot platforms. The application layer provides a suite of command line tools and a Web App to significantly lower the entry barrier for BotSIM users such as bot admins or practitioners. In this report, we focus on the technical designs of various system components. A detailed case study using Einstein BotBuilder is also presented to show how to apply BotSIM pipeline for bot evaluation and remediation. The detailed system descriptions can be found in our system demo paper. The toolkit is available at: https://github.com/salesforce/BotSIM .

我们介绍了Lavis，这是一个开源深度学习库，用于语言视觉研究和应用。拉维斯（Lavis）的目标是作为一个一站式综合图书馆，它为研究人员和从业人员提供了可访问语言视觉领域的最新进步，并赋予未来的研究和发展。它具有统一的界面，可轻松访问最新的图像语言，视频语言模型和常见数据集。 Lavis支持对各种任务的培训，评估和基准测试，包括多模式分类，检索，字幕，视觉问题答案，对话和预训练。同时，该库还高度可扩展且可配置，从而促进了未来的开发和定制。在此技术报告中，我们描述了图书馆的设计原理，关键组成部分和功能，并在常见的语言视觉任务中提出基准测试结果。该库可在以下网址获得：https：//github.com/salesforce/lavis。

在过去的十年中，以数据为驱动的特别基于深度学习的方法已成为机器人Grasp计划的主要范式。但是，这些方法的性能在很大程度上受到可用培训数据集质量的影响。在本文中，我们提出了一个框架来生成对象形状以增强握把数据集，从而可以提高预设计的深神经网络的掌握能力。首先，使用编码器解码器结构网络将对象形状嵌入到低维特征空间中。然后，使用异常检测和掌握质量标准计算每个对象形状的稀有性和掌握得分。最后，在特征空间中生成了新的对象，以利用原始的高稀有性和掌握分数对象的特征。实验结果表明，通过生成的对象形状可以有效提高基于学习的GRASP计划网络的掌握能力。

视频和语言预培训表明对各种下游任务有望改善。最先前的方法捕获与基于变换器的多模式编码器的跨模型交互，不完全解决单向视频和文本特征之间的错位。此外，学习细粒度的视觉语言对准通常需要离上的对象检测器来提供对象信息，这是由检测器有限的词汇和昂贵的计算成本的瓶颈。我们建议对齐和提示：一种高效有效的视频和语言预训练框架，具有更好的跨模型对齐。首先，我们介绍了一个视频文本对比（VTC）丢失，以对准实例级别的单峰视频文本功能，从而缓解跨模型交互的建模。然后，我们提出了一种新的视觉接地预训练任务，提示实体建模（PEM），旨在学习细粒度的区域实体对齐。为实现这一目标，我们首先介绍一个实体发射模块，该模块用VTC培训，以产生与实体名称实例化的视频裁剪和文本提示之间的相似性。 PEM任务然后询问模型以预测随机选择的视频作物的实体伪标签（I.E〜归一化相似度分数）。由此产生的预先训练的模型在文本 - 视频检索和VideoQ上实现了最先进的性能，通过大幅度的边距表现优于现有的工作。我们的代码和预先训练的型号将被释放。

尽管对象检测方面取得了很大进展，但由于实例级边界盒注释所需的巨大人性化，大多数现有方法都仅限于一小一少量的对象类别。为了减轻问题，最近的开放词汇和零射击检测方法试图检测培训期间未见的对象类别。但是，这些方法仍然依赖于一组基类上手动提供的边界盒注释。我们提出了一个开放的词汇检测框架，可以在没有手动提供边界盒注释的情况下培训。我们的方法通过利用预先训练的视觉语言模型的本地化能力来实现这一目标，并产生可直接用于训练对象探测器的伪边界盒标签。 Coco，Pascal VOC，Objects365和LVIS的实验结果证明了我们方法的有效性。具体而言，我们的方法优于使用人类注释的边界箱训练的最先进（SOTA），即使我们的培训源未配备手动边界盒标签，也可以在COCO新型类别上用3％AP培训。在利用手动边界箱标签作为基线时，我们的方法主要超过8％的AP。

Large-scale vision and language representation learning has shown promising improvements on various vision-language tasks. Most existing methods employ a transformer-based multimodal encoder to jointly model visual tokens (region-based image features) and word tokens. Because the visual tokens and word tokens are unaligned, it is challenging for the multimodal encoder to learn image-text interactions. In this paper, we introduce a contrastive loss to ALign the image and text representations BEfore Fusing (ALBEF) them through cross-modal attention, which enables more grounded vision and language representation learning. Unlike most existing methods, our method does not require bounding box annotations nor high-resolution images. To improve learning from noisy web data, we propose momentum distillation, a self-training method which learns from pseudo-targets produced by a momentum model. We provide a theoretical analysis of ALBEF from a mutual information maximization perspective, showing that different training tasks can be interpreted as different ways to generate views for an image-text pair. ALBEF achieves state-of-the-art performance on multiple downstream visionlanguage tasks. On image-text retrieval, ALBEF outperforms methods that are pre-trained on orders of magnitude larger datasets. On VQA and NLVR 2 , ALBEF achieves absolute improvements of 2.37% and 3.84% compared to the state-ofthe-art, while enjoying faster inference speed. Code and models are available at https://github.com/salesforce/ALBEF.

这封信涉及地形重建未知障碍环境的3D报道路径规划（CPP）问题。由于感测局限性，所提出的方法称为CT-CPP，执行3D区域的分层扫描以收集地形数据，其中使用覆盖树（CT）的概念与TSP启发的树遍历遍历策略进行优化。。CT-CPP方法在高保真水下模拟器上验证，结果与CPP方法后的现有地形进行了比较。结果表明，CT-CPP在轨迹长度，能量消耗和重建误差产生显着降低。